Experimental results about the assessments of conditional rank correlations by experts: Example with air pollution estimates

Science-based models often involve substantial uncertainty that must be quantified in a defendable way. Shortage of empirical data inevitably requires input from expert judgment. How this uncertainty is best elicited can be critical to a decision process, as differences in efficacy and robustness of the elicitation methods can be substantial. When performed rigorously, expert elicitation and pooling of experts’ opinions can be powerful means for obtaining rational estimates of uncertainty. Causes of uncertainty may be interrelated and may introduce dependencies. Ignoring these dependencies may lead to large errors. Dependence modelling is an active research topic, and methods for dependence elicitation are still very much under development. Dependence measures such as rank correlations are commonly used in different types of models. Eliciting rank correlations and conditional rank correlations from experts have been proposed and used in the past. Conditional rank correlations are not elicited directly from experts, rather the experts are asked to estimate some other related quantities. In this paper two methods for eliciting conditional rank correlations via related quantities are compared in order to obtain insight about which of the two renders more accurate estimates of conditional rank correlations. Our data shows that good performance in uncertainty assessments does not automatically translates into good performance in dependence estimates. We show that, analogously to uncertainty estimates, combining experts’ estimates of dependence according to their performance results in better estimates of the dependence structure. estimates of some other quantity, for example a conditional probability of exceedance or probabilities of concordance or discordance, and use these to estimate rank correlations (under certain copula assumptions). Though not conclusive, previous results indicate that the most accurate way to obtain a subjective measure of bivariate dependence is simply to ask the expert to estimate the correlation between the two variables in question (Clemen & et al., 2000). Recently, Non-Parametric Bayesian Networks (NPBN) have been introduced as flexible tools for applications where dependence modeling is important. See for example Ale et al. (2007), Hanea & Ale (2009). The inputs for these models are univariate marginal distributions and rank and conditional rank correlations. When field data is not available, rank and conditional rank correlations have been assessed from experts through the elicitation of Conditional Probabilities of Exceedance